-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ConnectorExpression pushdown #7994
Conversation
Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Assaf Bern.
|
core/trino-spi/src/main/java/io/trino/spi/expression/Function.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/expression/Function.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/expression/Function.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/expression/Function.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Show resolved
Hide resolved
hi @assaf2 are you still planning to work on this PR? this would be very useful for my organization for being able to push down more complex queries into postgres. |
@alec-heif good to hear, yes I'm on it |
8dadc26
to
c1c043a
Compare
@martint do you have a doc or some writeup outlining your thinking about desired end-goal for Besides the end-goal I would love to help us define intermediate steps that will allow us split the work into smaller pieces, without boiling the ocean. Defining those intermediate steps should not only help reach the end-goal sooner, but also would allow deliver partial value to users as we go. cc @losipiuk |
Greetings, |
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionPredicateTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionPredicateTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionPredicateTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionPredicateTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Show resolved
Hide resolved
plugin/trino-memory/src/main/java/io/trino/plugin/memory/MemoryMetadata.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/expression/ConnectorExpressionVisitor.java
Outdated
Show resolved
Hide resolved
plugin/trino-memory/src/test/java/io/trino/plugin/memory/TestMemorySmoke.java
Outdated
Show resolved
Hide resolved
cecbdc7
to
904867a
Compare
Hi everyone, may I know where we are with this PR? |
core/trino-main/src/main/java/io/trino/SystemSessionProperties.java
Outdated
Show resolved
Hide resolved
int dfaStatesLimit = SystemSessionProperties.getRe2JDfaStatesLimit(session); | ||
int dfaRetries = SystemSessionProperties.getRe2JDfaRetries(session); | ||
Re2JRegexp re2JRegExp = new Re2JRegexp(dfaStatesLimit, dfaRetries, slice); | ||
Type type = new Re2JRegexpType(dfaStatesLimit, dfaRetries); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Obtain the type object from TypeManager
via getType(Re2JRegexpType.RE2J_REGEXP_SIGNATURE)
instead of constructing an instance manually.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've tried to add TypeManager
to PlanOptimizers
's @Inject
constructor and to pass it from there. I had to change ~15 files including tests, but then I got a runtime error since PlanOptimizers
is bound at CoordinatorModule
, while TypeManager
is not bound yet.
I couldn't find another way to obtain TypeManager
, any suggestions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is what we need to implement this more generically, without having to hardcode any knowledge about how regexp functions work:
protected Optional<Expression> translateCall(Call call)
{
if (LIKE_FUNCTION_NAME.equals(call.getName())) {
return translateLike(call.getArguments().get(0), call.getArguments().get(1));
}
QualifiedName name = QualifiedName.of(call.getName());
FunctionCallBuilder builder = new FunctionCallBuilder(metadata)
.setName(name);
List<TypeSignature> argumentTypes = call.getArguments().stream()
.map(argument -> argument.getType().getTypeSignature())
.collect(toCollection(ArrayList::new));
ResolvedFunction resolved = metadata.resolveFunction(name, TypeSignatureProvider.fromTypeSignatures(argumentTypes));
for (int i = 0; i < call.getArguments().size(); i++) {
Expression expression = ConnectorExpressionTranslator.translate(session, call.getArguments().get(i), metadata, variableMappings, literalEncoder);
Type expectedType = resolved.getSignature().getArgumentTypes().get(i);
Type actualType = metadata.getType(argumentTypes.get(i));
if (!actualType.equals(expectedType)) {
verify(typeCoercion.canCoerce(actualType, expectedType));
expression = new Cast(expression, toSqlType(expectedType));
}
builder.addArgument(expectedType, expression);
}
return Optional.of(builder.build());
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
I just made a single change to your suggestion - replaced if (!actualType.equals(expectedType)) {
with if (!typeCoercion.isTypeOnlyCoercion(actualType, expectedType)) {
because, for example, at
testCrossJoinLargeBuildSideDynamicFiltering()
we get $internal$dynamic_filter_function
with a varchar(9)
argument while its expectedType = varchar
. Is this change ok by you?
In addition, are you aware of an equivalently generic way to implement io.trino.sql.planner.ConnectorExpressionTranslator.SqlToConnectorExpressionTranslator#visitFunctionCall
which still refers to JoniRegexp
and Re2JRegexp
specifically?
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
List<Expression> arguments = call.getArguments().stream() | ||
.map(argument -> ConnectorExpressionTranslator.translate(session, argument, metadata, literalEncoder, Optional.empty())) | ||
.collect(Collectors.toList()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This translation can be done inside the if
block below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The translator currently supports many functions.
If I'll do so, I'll need to create a FunctionCall
only inside the if
, so only regexp_like
and LIKE
functions will be supported. Therefore, I'll also need to change io.trino.sql.planner.ConnectorExpressionTranslator.SqlToConnectorExpressionTranslator#visitFunctionCall
to convert only regexp functions (because otherwise we'll translate only in one direction).
Do you want me to narrow the translation this way?
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
This for the hard work, not sure if #9067 belongs to this issue? |
904867a
to
2cbf2a0
Compare
82d9db9
to
3d6ce2e
Compare
7d9ae24
to
30e2310
Compare
core/trino-main/src/main/java/io/trino/sql/planner/ConnectorExpressionTranslator.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/expression/FunctionName.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/expression/FunctionName.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/expression/FunctionName.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/main/java/io/trino/spi/expression/FunctionName.java
Outdated
Show resolved
Hide resolved
@@ -249,14 +269,24 @@ private boolean arePlansSame(FilterNode filter, TableScanNode tableScan, PlanNod | |||
node.isUpdateTarget(), | |||
node.getUseConnectorNodePartitioning()); | |||
|
|||
Expression remainingDecomposedPredicate; | |||
if (connectorExpression.isEmpty() || remainingConnectorExpression.isEmpty() || remainingConnectorExpression.equals(connectorExpression)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If connectorExpression
is empty (i.e., either there wasn't a expression in the first place or the translation from Expression->ConnectorExpression failed), it is still possible that remainingConnectorExpression
is not empty (i.e., if the connector synthesizes an expression). In that case, we need to combine decomposedPredicate.getRemainingExpression()
with the result from applyFilter()
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Modified
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
( followed up here: #7994 (comment) )
30e2310
to
f121df1
Compare
f121df1
to
0835585
Compare
core/trino-main/src/main/java/io/trino/sql/ExpressionUtils.java
Outdated
Show resolved
Hide resolved
...trino-main/src/main/java/io/trino/sql/planner/iterative/rule/PushPredicateIntoTableScan.java
Outdated
Show resolved
Hide resolved
remainingDecomposedPredicate = ExpressionUtils.combineConjuncts(plannerContext.getMetadata(), translatedExpression, decomposedPredicate.getRemainingExpression()); | ||
} | ||
else { | ||
remainingDecomposedPredicate = translatedExpression; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The conversion
Optional<ConnectorExpression> connectorExpression = new ConnectorExpressionTranslator.SqlToConnectorExpressionTranslator(session, remainingExpressionTypes, plannerContext)
.process(decomposedPredicate.getRemainingExpression());
is lossy.
i.e. decomposedPredicate.getRemainingExpression()
contains full information
but then connectorExpression
contains partial information
remainingConnectorExpression
contains part of connectorExpression
, so again not a full information
so then translatedExpression
cannot supersede remainingDecomposedPredicate
i think SqlToConnectorExpressionTranslator
should work like a DomainTranslator
, i..e it should spit out (a) translated result and (b) remaining expression
@martint do you have a suggestion how to handle this for now, so that SqlToConnectorExpressionTranslator
change can be a follow up?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is lossy.
That should not be the case. The translation should either succeed and be equivalent to the original expression, or fail and return Optional.empty().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
core/trino-main/src/main/java/io/trino/sql/ExpressionUtils.java
Outdated
Show resolved
Hide resolved
1978d9f
to
110c2ce
Compare
Squashed fixups. Previous CI run https://github.com/trinodb/trino/actions/runs/1909288734 has some failures but apparently not related. let's see CI this time. |
110c2ce
to
a96f5ee
Compare
Merged, thanks! |
Really excieted to see this PR merged. |
Optional<Expression> translatedValue = translate(session, value); | ||
Optional<Expression> translatedPattern = translate(session, pattern); | ||
if (translatedValue.isPresent() && translatedPattern.isPresent()) { | ||
return Optional.of(new LikePredicate(translatedValue.get(), translatedPattern.get(), Optional.empty())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For #18